Method and system for re-uniting metadata with media-stream content at a media client, to facilitate action by the media client

ABSTRACT

In a communication system where a channel teed carrying a media stream is broadcast for delivery of the media stream to a media client, where the channel feed further includes metadata in-band and correlated with a content frame of the media stream, but where the metadata gets removed from the channel feed before the media stream reaches the media client, a computing system (i) extracts the metadata from the channel feed and (ii) transmits out-of-band to the media client the extracted metadata in correlation with reference fingerprint data representing the frame of the media stream, the correlation being useable by the media client as a basis to re-correlate the metadata with the frame of the media stream, so as to enable the media client to carry out an action, such as dynamic ad insertion, based on the correlation of the metadata with the frame of the media stream.

REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. non-provisional patentapplication Ser. No. 17/447,711, filed Sep. 15, 2021, which claimspriority to U.S. non-provisional patent application Ser. No. 15/929,211,filed Jan. 28, 2020, which claims priority to U.S. Provisional PatentApplication No. 62/900,043, filed Sep. 13, 2019, the entirety of each ofwhich is hereby incorporated by reference.

BACKGROUND

A typical media client operates to receive an analog or digital mediastream representing media content such as video and/or audio content andto process the media stream for presentation by the media client or anassociated entity on a user interface such as a display screen and/or anaudio speaker. Examples of such media clients could include televisions,computer monitors, projection systems, loudspeakers, headphones, set topboxes (e.g. cable or satellite TV receivers), digital video recorders,radios, personal computers, mobile communication devices, gamingconsoles, streaming media players, and/or combinations of these otherdevices.

By way of example, a television could receive a broadcast stream (e.g.,over the air, from a set top box, through an Internet connection, or inanother manner) and could present the media content of that broadcaststream to a user. As another example, a set top box could receive abroadcast stream from a Multi-channel Video Program Distributor (MVPD)and could output the media content of that stream via a. High-DefinitionMultimedia. Interface (HDMI) cable or other interface to a televisionand/or audio/video receiver for playout. As still another example, adigital video or audio recorder could likewise receive a broadcaststream but could store the broadcast stream for later playout. And asyet another example a loudspeaker or headphones could receive abroadcast audio stream from a radio, computer, or other device, andcould present the audio content of that stream to a user. Numerous otherexamples are possible as well.

SUMMARY

The present disclosure provides a technical advance that could help toaddress a situation in which a channel feed carrying a media stream isbroadcast for delivery of the media stream to a media client, where thechannel teed further includes metadata in-band and correlated with acontent frame of the media stream, but where the metadata gets removedfrom the channel feed before the media stream reaches the media client.

In accordance with the disclosure, a computing system could extract themetadata from the channel feed and could then transmit in an out-of-bandmessage to the media client the extracted metadata in correlation withreference fingerprint data representing the frame of the media stream.The correlation of the extracted metadata with the reference fingerprintdata representing the frame of the media stream could then be useable bythe media client as a basis to re-correlate the metadata with the frameof the media stream, so as to enable the media client to carry out anaction based on the correlation of the metadata with the frame of themedia stream.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description, with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescriptions provided in this summary and below are intended toillustrate the invention by way of example only and not by way oflimitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of an example system in whichvarious disclosed principles can be applied.

FIG. 2 is a simplified block diagram depicting an example process thatcould be could be carried out in accordance with the present disclosure.

FIG. 3 is another simplified block diagram depicting an example processthat could be could be carried out in accordance with the disclosure.

FIG. 4 is a simplified block diagram of an example computing system thatcould operate in accordance with the disclosure.

FIG. 5 is a simplified block diagram of an example media client thatcould operate in accordance with the disclosure.

FIG. 6 is a flow chart depicting a method that could be carried out inaccordance with the disclosure.

DETAILED DESCRIPTION

Referring to the drawings, as noted above, FIG. 1 is a simplified blockdiagram of an example system in which various disclosed principles canbe applied. It will be understood, however, that this and otherarrangements and processes described herein can take various otherforms. For instance, elements and operations can be re-ordered,distributed, replicated, combined, omitted, added, or otherwisemodified. Further, it will be understood that functions described hereinas being carried out by one or more entities could be implemented byand/or on behalf of those entities, through hardware, firmware, and/orsoftware, such as by one or more processing units executing programinstructions or the like.

As shown in FIG. 1 , the example system includes one or more mediacontent sources 100 (e.g., broadcasters, web servers, etc.), one or moremedia content distributors 102 (e.g., MVPDs, such as cable providers,satellite providers, over-the-air broadcast providers, web aggregators,etc.), one or more media content receivers 104 (e.g., cable receivers,satellite receivers, over-the-air broadcast receivers, computers orother streaming media receivers, etc.), and one or more clients orcontent presentation devices 106 (e.g., TVs or other display devices,loudspeakers or other audio output devices, etc.)

In an example implementation, the media content sources 100 could benational broadcasters, such as ABC, NBC, CBS, FOX, HBO, and CNN, themedia content distributors 102 could be local affiliates and/or otherlocal content distributors in designated market areas (MIAs), and thereceivers 104 and content presentation devices 18 could then be situatedat customer premises, such as homes or business establishments. Withthis or other arrangements, the content sources 100 could deliver mediacontent to the content distributors 102 for distribution to receivers104 at customer premises, and the content distributors could distributethe media content to the receivers 104 on discrete channels (e.g.,particular frequencies or other defined channels). Each receiver couldthen respond to user input or one or more other triggers by tuning to aselected channel and outputting to a content presentation device 106 themedia content that is arriving on the selected channel. And the contentpresentation device 106 could receive and render the media content(e.g., display or otherwise present the content).

When a media client such as a receiver 104 and/or content presentationdevice 106 is processing a linear media stream for presentation (e.g.,as the media client is receiving the media stream, rendering the mediastream for presentation, outputting the media stream for presentation,and/or otherwise processing the media stream for presentation), it maybe useful in certain situations for the media client to perform aspecial content-related action, i.e., an action related to the contentof the media stream.

By way of example, the content-related action could involve initiatingan interactive event, such as beginning to superimpose overlay content(e.g., overlay audio and/or video content) that prompts a user torespond in some manner or that otherwise relates to interaction with auser of the media client or of an associated user interface on whichcontent of the media stream is presented.

And as another example, the content-related action could involvedynamically revising a portion of the stream's media content. Forexample, it may be useful for the media client to dynamically replace anad or other replaceable segment of the media stream with a replacementad or with other replacement content, so that when the content is playedout to a user, the user would receive the replacement ad or otherreplacement content in place of the originally included ad or otherreplaceable segment. As another example, it may be useful for the mediaclient to dynamically supplement a portion of the media content withoverlay or split-screen content, such as channel identification, contextinformation, ad content, or the like, so that when the content is beingplayed out to a user, the user would receive the supplemental contenttogether with the originally included content.

Further, it may be desirable for the media client to perform thecontent-related action at a specific time point within the media stream.For instance, if the media client is to dynamically replace an existingad or other replaceable segment of the media stream with a replacementad or other replacement content, it may be desirable for the mediaclient to position the replacement ad or other replacement content at atime in the media stream when the existing ad or other replaceablesegment would have started. Likewise, if the media client is tosupplement a specific portion of the media content with an overlayrelated to that portion, it may be desirable for the media client tostart the overlay when that portion starts. Other examples are possibleas well.

More particularly, the media stream at issue could define a sequence offrames, such as video frames and/or audio frames, and it may bedesirable for the media client to perform the content-related action ona frame-accurate basis. For instance, for dynamic content replacement,it may be desirable for the media client to insert the replacement ad orother replacement content in place of the existing ad or otherreplaceable segment starting precisely at the first frame of theexisting ad or other replaceable segment. And for adding an overlay to aspecific portion of the media stream, it may be desirable for the mediaclient to start the overlay at or in specific relation to the firstframe of that portion of the media stream. Similar rationale may existfor other of content-related actions as well.

To facilitate this, an entity that provides the media stream couldinclude in-band with the media stream certain metadata related to theupcoming content-related-action opportunity, such as metadata specifyingtiming of the upcoming content-related action and/or metadatacharacterizing the type of content-related action to perform, amongother possibilities. When the media client receives the media streamtogether with such metadata, the media client could then use themetadata as a basis to carry out the content-related action.

This metadata could be included in-band with the media stream in variousways, and the media client could accordingly extract the in-bandmetadata in various ways to facilitate use of the metadata for carryingout the content-related action. Including the metadata in-band with themedia stream involves providing the metadata with the media stream, suchas in the same communication as the media stream, so that the metadatawould be received with the media stream. This is to be distinguishedfrom providing the metadata out-of-band, where the metadata is providedseparately from the media stream, such as in a separate communication,possibly through a different communication channel, for instance.

One way to provide metadata in-band with a media stream is to encode themetadata into the media stream itself. For instance, the metadata couldbe steganographically encoded as watermarking (e.g., one or more audioor video watermarks) in content of the media stream, using any ofvarious watermarking techniques now known or later developed. In thatcase, the media client could evaluate the incoming media stream todetect the presence of such a watermark and, upon detecting thewatermark, could decode the watermark and thus read the metadata fromthe media stream. Alternatively, the metadata could be otherwise encodedin the media stream, such as within a vertical ancillary data (VANC)space of a video frame for instance, in which case the media clientcould read the metadata from the VANC, space of the video frame, amongother possibilities.

And as another example, if the media stream is carried in a transportstream (e.g., in a packet-data stream), then the transport stream mightcarry the metadata within the header of each of headers of one or moretransport-stream packets and/or within one or more special packets ofthe transport stream. In that case, the media client could evaluate theincoming transport stream carrying the media stream to detect thepresence of the metadata at the transport-stream layer and could thenread the metadata from the transport stream.

These or other forms of in-band metadata could be provided using variousprotocols, such as but not limited to those defined by standardsorganizations such as the Society of Cable and TelecommunicationsEngineers (SCTE) (e.g., SCTE 104 or SCTE 35), the Advanced TelevisionSystems Committee (ATSC) ATSC 3.0), or the European TelecommunicationsStandards Institute (e.g., Hybrid Broadcast Broadband (HBB) TV (IS 102796)), among other possibilities.

As is known in the art, for instance, watermarking involves permanentlyembedding or otherwise encoding information into media content in amanner that enables the information to be decoded and extracted from themedia content by a receiver of the media content but that may beimperceptible to a user to whom the media content is presented. This isin contrast to inserting such information into a packet header or thelike without encoding the information into the underlying media contentitself. Watermarking permanently changes the media content and may beimpossible to remove.

More particularly, watermarking media content could involve encodinginto the media content a code that can be mapped to associatedinformation, or perhaps more directly encoding into the media contentthe associated information. In an example implementation, the watermarkcode could be on the order of 24 bits, and the watermarking could bedone in an audio component of the media content and/or in a videocomponent of the media content, depending on the form of the mediacontent for instance.

Existing audio watermarking techniques include, without limitation,inserting audio energy into the audio signal or otherwise adjusting oneor more characteristics of the audio signal in a manner that can bedetected by a suitably configured watermark-detection algorithm but thatis masked from hearing (e.g., the audio characteristic and/or manner ofencoding being sufficient to hide the watermark from humandetection)—known as stenographic or psychoacoustic encoding.

Examples of audio watermarks and corresponding watermark detectiontechniques are described in U.S. Pat. No. 8,359,205 (entitled “Methodsand Apparatus to Perform Audio Watermarking and Watermark Detection andExtraction,” which issued on Jan. 22, 2013), U.S. Pat. No. 8,369,972(entitled “Methods and Apparatus to Perform Audio Watermarking Detectionand Extraction,” which issued on Feb. 5, 2013), U.S. Patent ApplicationPub. No. 2010/0223062 (entitled “Methods and Apparatus to Perform AudioWatermarking and Watermark Detection and Extraction,” which waspublished on Sep. 2, 2010), U.S. Pat. No. 6,871,180 (entitled “Decodingof Information in Audio Signals,” which issued on Mar. 22, 2005), U.S.Pat. No. 5,764,763 (entitled “Apparatus and Methods for including Codesin Audio Signals and Decoding,” which issued on Jun. 9, 1998), U.S. Pat.No. 5,574,962 (entitled “Method and Apparatus for Automaticallyidentifying a. Program Including a Sound Signal,” which issued on Nov.12, 1996), U.S. Pat. No. 5,581,800 (entitled “Method and Apparatus forAutomatically Identifying a Program including a Sound Signal,” whichissued on Dec. 3, 1996), U.S. Pat. No. 5,787,334 (entitled “Method andApparatus for Automatically Identifying a Program Including a SoundSignal,” which issued on Jul. 28, 1998), and U.S. Pat. No. 5,450,490(entitled “Apparatus and Methods for Including Codes in Audio Signalsand Decoding,” which issued on Sep. 12, 1995, all of which are herebyincorporated by reference in their entireties.

Existing video watermarking techniques, on the other hand, involveembedding a code in a video component of the media content in a mannerthat can be detected by a suitably configured watermark-detectionalgorithm but that is masked from human visual detection.

Examples of video watermarking techniques include various spatial-domaintechniques such as flipping pixels, embedding the watermark into leastsignificant bits, and adding a pseudo-random noise pattern to the video,and various frequency-domain techniques, such as SVD domainwatermarking, Discrete Fourier Transform watermarking, Discrete CosineTransform watermarking, Discrete Wavelet Transform watermarking, andprincipal component analysis watermarking. Other examples are possibleas well.

Metadata provided in-band using these or other techniques could beprovided with the media stream in a manner that correlates the metadatawith an associated point in the media stream, such as a particular frameamong the sequence of frames defining the media stream. By way ofexample, the metadata could be watermarked in a specific frame of themedia stream to correlate the metadata with that specific frame. Or themetadata might be provided in a transport packet carrying or adjacent insequence to a specific frame of the media stream to correlate themetadata with that specific frame. Alternatively, the in-band metadatamight be correlated with a particular frame or other portion of themedia stream in another manner.

With this or a similar arrangement, the in-band metadata could providethe media client with various useful information to enable the mediaclient to carry out a content-related action.

By way of example, if the content-related action is dynamic ad insertion(DAI), where the media client would replace playout of a replaceable adin the media stream with playout of a replacement ad (such as onetailored to user demographics), the metadata could provide informationthat informs the media client when in the media stream the replaceablead starts, so that the media client could start inserting thereplacement ad at the frame-accurate point during the media client'sprocessing of the media stream. For instance, the metadata could simplybe an advertisement identifier (Ad-ID) provided in-band in correlationwith at least one or more starting frames of the replaceable ad, so thatthe media client could detect the in-band metadata as an indication ofthe frame where the media client should start inserting the replacementad.

As another example, the in-band metadata provided in correlation with aparticular frame or other time point in the media stream could specify atime delta that informs the media client how far ahead from that pointin the media stream the media client should start performing thecontent-related action. For instance, the in-band metadata provided incorrelation with a particular frame of the media stream might direct themedia client to insert a replacement ad starting N frames later thanthat particular frame or N milliseconds later than that particularframe, among other possibilities.

As still another example, the in-band metadata could cause the mediaclient to carry out any of a variety of other content-related actions,such as to apply a graphical overlay to the content of the media stream,and/or to initiate presentation of a specified application such as aninteractive program, presentation of e-commerce: related opportunitiespossibly including product placement, offers of coupons, and/orpresentation of personalized content recommendations such as contentsimilar to the content being presented, among other possibilities.

One technical problem that can occur with use of this or other suchin-band metadata is that the in-band metadata might be stripped orotherwise dropped when the media stream is en route to the media client,so the media client may receive the media stream without receiving themetadata and may therefore not carry out the associated content-relatedaction.

By way of example, as a media stream with the in-band metadata is beingcommunicated along a media-distribution path en route to the client, anentity within the media-distribution path might receive the media streamand the metadata, and that entity might strip the metadata and forwardthe media stream without the metadata along the media-distribution pathfor ultimate receipt by the media client.

For instance, in an implementation where the media client is a contentpresentation device such as a TV, a customer-premises receiver upstreamfrom the media client might receive from an MVPD or other entity atransport stream that carries a media stream representing a channel ofcontent and that also carries metadata. As the receiver receives thattransport stream, the receiver may extract the media stream from thetransport stream (e.g., de-packetize the media stream) and output theextracted media stream through HDMI or another interface to the mediaclient for presentation. But in this process, the receiver may notforward to the media client the metadata that was also included in thetransport stream. Therefore, the media client may not receive themetadata that was included in the transport stream and so may be unableto carry out a content-related action based on that netadata.

As another example, an entity within the media-distribution path mightreceive the transport stream that carries the media stream with themetadata and might strip the metadata from the transport stream beforeforwarding the transport stream along the media-distribution path, sothat a downstream entity receiving the media stream would likewise notreceive the metadata that was carried by the transport stream.

For instance, the SCTE 35 standard provides for inclusion in a transportstream of splice_information packets that specify when dynamic contentmodification should occur in the media stream carried by the transportstream. However, industry practice in accordance with the SCTE 35standard suggests that a content distributor (e.g., a terrestrialover-the-air broadcaster, or a satellite or cable MVPD) should strip anysplice_information packets from a transport stream before delivering thetransport stream to customer premises equipment, in order to preventunauthorized “commercial killers” or the like. In particular, theindustry concern is that an unauthorized party could detect the presenceof splice_information packets and could undesirably remove (e.g., skippast) or replace included commercials in the media stream. Althoughsplice_information packets could be secured through scrambling and/orencryption, the packets may still be sent separately within thetransport stream, and there is a risk that they could be detected andused without authorization. Consequently, it may not be feasible orpractical to maintain splice_information packets in a transport streamen route to an end-user media client, and so the end-user media clientmay be unable to use that in-band metadata as a basis to carry outdynamic content modification.

Further, similar loss of in-band metadata could occur with respect toother types of in-band metadata, such as metadata provided in the VANCspace or through watermarking, among other possibilities.

Considering this technical issue, at issue is then how to convey to themedia client the metadata that was included in-band with the mediastream, and particularly how to convey that metadata to the media clientin a manner that would properly correlate or otherwise reunite themetadata with the frame or other portion of the media stream that themetadata was correlated with in the first place.

In accordance with the present disclosure, a computing system willextract the metadata that is included in-band with the media stream enroute to the media client and will then transmit the extracted metadataout-of-band to the media client in a manner that enables the mediaclient to correlate the metadata with the particular frame or otherportion of the media stream with which the in-band metadata wascorrelated.

For instance, the computing system could transmit the extracted metadataover a broadband network connection to the media client, in message thatincludes data correlating the extracted metadata with the particularposition within the media stream where the in-band metadata wasprovided.

To facilitate this the computing system and media client couldcommunicate with each other over a broadband network, which could be apacket-switched network such as the Internet. For instance, the mediaclient could sit as a node on a local area network (LAN) at customerpremises, with the media client having an assigned Internet Protocol(IP) address on the LAN and the LAN having an IP address on thebroadband network. And the computing system could likewise have an IPaddress enabling the computing system to communicate on the broadbandnetwork, such as to provide an out-of-band communication to the mediaclient. The computing system could thus transmit the extracted metadatato the media client in a message according to any agreed communicationprotocol over the broadband network, among other possibilities.

In an example implementation, the computing system could transmit theextracted metadata out-of-band to the media client in correlation with aparticular portion of the media stream by transmitting the extractedmetadata to the media client in correlation with one or more digitalreference fingerprints that represent one or more particular frames ofthe media stream. For instance, the computing system could generate orobtain the digital reference fingerprints representing each sequentialframe of the media stream that is en route to the media client, and thecomputing system could transmit those reference fingerprints to themedia client on an ongoing basis along with the metadata correlated toparticular ones of the reference fingerprints representing the frameswith which the metadata was correlated when the metadata was in-band.

Without limitation, digital fingerprinting with respect to video couldapply on a per video frame basis and could involve establishing arepresentation of luminosity and/or other video characteristics. Forinstance, for a given video frame, a fingerprint generator couldprogrammatically divide the frame into a grid, and the fingerprintgenerator could measure luminosity of the frame per grid cell andgenerate a bit string with each bit or series of bits representingluminosity of a respective grid cell, or representing a weighteddifference between the luminosity of certain defined pairs of the gridcells, or the like. Further, the fingerprint generator could apply thisprocess continually to generate the digital fingerprint over time as asequence of fingerprints (as a fingerprint stream). For instance, thefingerprint generator could apply this process to each frame, to eachkey frame, periodically, or on another defined basis, with each frame'sbit string defining a digital fingerprint and/or with a specified hash,combination or series of such bit strings or other representative valuesdefining a digital fingerprint, on a sliding window basis. Other digitalfingerprinting mechanisms could be used as well, further including audiofingerprinting.

In a representative implementation, the computing system could correlatea reference fingerprint with particular extracted metadata by attachingthe metadata to the reference fingerprint in a data relationship thatwould be understandable to the media client. For instance, the computingsystem could establish a data record that contains both the referencefingerprint and the metadata and could send the data record to the mediaclient, so that the media client would receive the reference fingerprintand the associated metadata together. And if the computing system issending a sequence of reference fingerprints to the media client, thecomputing might send a sequence of such data records to the mediaclient. As a related example, the computing system might send eachreference fingerprint to the media client in the payload portion of apacket, and the computing system might provide the associated extractedmetadata, if any, in a header of that packet.

Alternatively, the computing system could use pointers or other data asa basis to correlate reference fingerprints with the particularextracted metadata.

For example, the computing system could use timestamps as a basis tocorrelate the reference fingerprints with the extracted metadata. In atleast this implementation, each of the reference fingerprints thatrepresents a frame of the media stream that is en route to the mediaclient could be timestamped with reference timestamp data indicating therelative time position of the frame represented by the referencefingerprint within the media stream, according to a timeline definedwithin the media stream or a clock of a content distributor or the like,for instance. Further, the computing system could likewise timestampeach instance of extracted metadata with a reference timestampindicating the relative time position of the frame with which themetadata was correlated when it was in-band, also according to the sametimeline. On an ongoing basis, the computing system could then deliverto the media client both the timestamped metadata and the timestampedreference fingerprints.

Provided with these references fingerprints in correlation with theextracted metadata, the media client could then conveniently reunite(re-correlate) the metadata with one or more frames or other portions ofthe media stream with which the metadata was correlated when it wasin-band.

To do this, the media client could generate digital query fingerprintsrepresenting individual frames of the media stream as the media clientis presenting the media stream. The media client could apply the samedigital fingerprinting process that is used to generate the referencefingerprints provided by the computing system. And the media clientcould compare those query fingerprints with the provided referencefingerprints in an effort to find fingerprint matches with sufficientcertainty.

To compare a given query fingerprint with a given reference fingerprint,the media client could compare corresponding portions of thefingerprints with each other to determine whether the portions matchexactly or within defined tolerances. For example, the media clientcould compute a maximum deviation between the fingerprints and determineif the maximum deviation is within a predefined tolerance. Further, ifthe fingerprints are binary, this could be a Boolean determination orcould involve computing a Hamming distance (as a count of mismatchesbetween respective bit locations in the fingerprints), and if thefingerprints are more complex values, such as decimal values or vectors(e.g., grey values per video frame region), this could involvedetermining a distance between the values or vectors. Other examples arepossible as well.

The media client could specifically conduct this fingerprint matchingfor at least any reference fingerprint that the computing systemprovided in correlation with extracted metadata. For each such referencefingerprint, the media client could seek to find a query fingerprintthat matches the reference fingerprint. And upon finding a match withsufficient certainty between a query fingerprint representing a frame ofthe media stream and a reference fingerprint provided by the computingsystem, the media client could reasonably conclude that the metadatathat the computing system provided in correlation with that referencefingerprint corresponds with the frame of the media stream at the mediaclient.

Upon so re-correlating the extracted metadata with the frame of themedia stream, the media client could then treat the metadata as thoughthe metadata has been provided expressly in correlation with that frameof the media stream. For instance, if the metadata includes a firstinstance of an Ad-ID representing a replaceable ad in the media stream,the media client may use that metadata as a basis to start inserting areplaceable ad at that frame in the course of the media client'sprocessing (e.g., playout) of the media stream. Or if the metadataincludes an indication of how far ahead in the media stream acontent-modification or other action is to occur, the media client couldprepare to carry out the action at that upcoming time point in relationto the time point where the metadata was reunited.

In a further implementation, the media client might add back theprovided metadata as in-band metadata for reference and use by an entitydownstream from the media client. For instance, if the media client is aset top box or other receiver that serves a content presentation devicesuch as a TV, the media client might receive the extracted metadata inthe manner described above, and the media client might thensteganographically encode that metadata into the media stream incorrelation with the associated frame(s) of the media stream. The mediaclient may then output the resulting media stream withsteganographically encoded metadata for receipt by the TV. This way, theTV could then decode and read the metadata from the media stream andcould take action based on where the metadata was encoded in the mediastream.

In the above processes, note that act of the computing system providingthe reference fingerprints to the media client could serve additionalpurposes as well.

For example, providing the reference fingerprints to the media clientmight facilitate establishing “synchronous lock” of time clocks betweenthe computing system and the media client, such as between areference-time clock used for reference-fingerprint timestamps at thecomputing system and a client-time clock used at the media client.

Establishing synchronous lock could be useful in a scenario where thecomputing system might seek to direct the media client to carry out acontent-related action at a particular time point in the media stream,but where the computing system and media client have differentperspectives for time. The computing system may characterize the timepoint in the media stream in the context of a reference timeline, suchas a timeline defined along the media stream and/or at a distributor ofthe media stream. Whereas, the media client may define that time pointin the media stream according to a client timeline, such as a timelinedefined according to a clock at the media client. This time discrepancycould be because of a broadcast time delay between a media distributor(e.g., over-the-air broadcaster or satellite or cable MVPD) and themedia client and/or if the media client is outputting the media streamwith time shifting, where the media client received and stored the mediastream for later playout and is now outputting the media stream fromstorage.

In this scenario, the computing system could provide the media clientwith the reference fingerprints timestamped according to the referencetimeline, and the media client could timestamp the query fingerprintsaccording to the client timeline. The media client could then performfingerprint matching as discussed above. And for each fingerprint matchthat the media client finds with sufficient certainty, the media clientcould compute a difference between the reference timestamp of thematching reference fingerprint and the client timestamp of the matchingquery fingerprint. The media client could then conclude that thatcomputed time difference represents the time offset between referencetime and client time, or the media client could conclude that a rollingaverage (or the like) of such computed time differences represents thetime offset between reference time and client time.

If the media client then receives from the computing system a directiveto carry out a content-related action at a time point in the mediastream characterized according to reference time, then the client couldapply the established time offset to translate that reference time pointto an associated client time point, and the media client could carry outthe content-related action at that client time point. Alternatively, themedia client could inform the computing system what time offset themedia client has computed, and the computing system could apply thattime offset to translate a reference time point to a client time pointand could direct the media client to carry out the content-relatedaction at that client time point.

Note also that, in order to facilitate carrying out the processesdiscussed herein, the computing system may need to initially identifythe media stream that the media client is presenting, so that thecomputing system could then obtain the reference fingerprints of thatparticular media stream to provide to the media client along with theextracted metadata. The computing system could identify in various waysthe media stream that the media client is presenting.

As one example, the media client might regularly generate and transmitto the computing system digital query fingerprints representing framesof the media stream that the media client is presenting. And thecomputing system might have access to reference fingerprint data thatincludes reference fingerprints representing each of various differentmedia streams that the media client could possibly be presenting.

In practice, the computing system might receive these referencefingerprints from one or more distributors of the media streams, withcapture engines operating at each distributor to generate the referencefingerprints as to each media stream being distributed, and with eachreference fingerprint being mapped with a unique identity of the mediastream whose frame it represents. Or the computing system might receivecopies of each such media stream being distributed and might itselfgenerate the reference fingerprints per media stream, with eachreference fingerprint likewise being mapped with a unique identity ofthe media stream whose frame it represents.

As the computing system receives the query fingerprints from the mediaclient, the computing system could then search through the referencedata for reference fingerprints that match the query fingerprints withsufficient certainty. And upon finding with sufficient certainty thatthe query fingerprints match reference fingerprints representing framesof a particular media stream, the computing system could reasonablyconclude that that is the media stream being presented by the mediaclient.

Once the computing system has thus identified the media stream beingpresented by the media client, the computing system could thenspecifically obtain the reference fingerprints representing thatparticular media stream. And the computing system could then proceedwith the above process, providing the media client with those referencefingerprints in correlation with metadata that the computing systemextracted, to enable the media client to reunite that metadata with theframes of the media stream with which the metadata was correlated whenthe metadata was in-band.

Further, note that it may be most efficient to carry out the aboveprocess of reuniting the extracted metadata with the correspondingframes or other portions of the media stream just in a scenario wherethe media client is not actually receiving the metadata—such as wherethe in-band metadata is being stripped or otherwise dropped en route tothe media client. To help ensure that the process is carried out in thisscenario, as an example implementation, the media client could checkwhether the media client is receiving the in-band metadata and couldinform the computing system one way or the other.

In an example implementation, for instance, the computing system couldsimply start to carry out the above process, including transmittingout-of-band to the media client the extracted metadata for reunitingwith particular frames of the media stream. And when the media clientreceives that out-of-band-transmitted metadata, the media client couldcheck to determine if the media client also receives the same metadatain-band with the media stream. If so, then the media client couldtransmit an out-of-band message to the computing system requesting thecomputing system to discontinue transmitting the extracted metadata tothe media client. Whereas, if not, then the media client could proceedto use the provided extracted metadata in the manner discussed above.

Alternatively, the media client could otherwise check to determine ifthe media client receives in-band metadata and could inform thecomputing system accordingly to control whether the computing systemwill go through the process of providing the media client with theextracted metadata. Here, the media client might expect to receivecertain in-band metadata and could check to see if it receives thatin-band metadata, so as to inform the computing system accordingly.

As a specific example of this, test or dummy metadata could be includedin-band with the media stream (e.g., by the media stream provider)specifically for purposes of determining whether the media clientreceives in-band metadata, and the computing system's extracting of thein-band metadata and providing of the extracted metadata to the mediaclient for reuniting with the associated frames of the media streamcould be conditioned on the media client not receiving the test or dummyin-band metadata. If the media client receives the test or dummy in-bandmetadata, then the media client could inform the computing system ofthat fact, so that the computing system would then not engage in theprocess of extracting and providing the metadata as discussed above.Whereas, if the media client does not receive the test or dummy in-bandmetadata, then the media client could inform the computing system ofthat fact, so that the computing system would then engage in the processof extracting and providing the metadata as discussed above.

FIGS. 2 and 3 are simplified block diagrams depicting how this processcould be carried out in two example scenarios, without limitation.

Each of these figures depicts an example arrangement where a linearchannel feed (e.g., a TV channel) passes from a broadcast playout source(broadcast facility) through broadcast infrastructure (e.g., an over-theair, cable) to a media client (e.g., a TV), possibly via a set top box.As shown, the broadcast playout source provides the channel feedtogether with in-band metadata provided at a transport-stream layer(e.g., SCTE 35 metadata) and/or with watermarking of the channel feeditself. As discussed above, each instance of this in-band metadata wouldbe provided in correlation with a particular content frame or other suchportion of the channel feed, Yet as further shown, the metadata may bestripped as it passes through the broadcast infrastructure or elsewherein the distribution path en route to the media client.

Each figure further shows a capture server that receives from thebroadcast playout source a copy of the channel feed with the in-bandmetadata. This capture server could be a physical device and processingengine at the broadcast facility and/or a remote watching station thatis configured to receive a copy of the channel feed with the in-bandmetadata.

The capture server operates to generate timestamped referencefingerprints of content frames of the channel feed, using any digitalfingerprinting process now known or later developed. Further, thecapture server operates to extract the in-band metadata provided withthe channel feed. And the capture server operates to correlate eachinstance of extracted metadata with the timestamped referencefingerprint representing a channel feed frame with which the metadata iscorrelated. For instance, the capture server could attach the metadatawith the reference fingerprint in a common data record or the like. Orthe capture server could timestamp the metadata with the same timestampas the reference fingerprint, so that the metadata and referencefingerprint are correlated by having the same timestamp as each other.

FIG. 2 depicts an implementation where the capture server correlateseach instance of metadata with the associated reference fingerprint bytimestamping the metadata and reference fingerprint with the sametimestamp as each other. This implementation includes by way of examplean automatic content recognition (ACR) service and a metadata service,which could be server applications executed on separate servers and/oron a common server, possibly integrated with the capture server. Withthis example arrangement, as the capture server generates thetimestamped reference fingerprints and timestamped metadata, the captureserver provides the timestamped reference fingerprints to the ACRservice and provides the timestamped metadata to the metadata service.

In this example arrangement, as the ACR service receives the oftimestamped reference fingerprints, the ACR service provides thetimestamped reference fingerprints to the media client. In line with thediscussion above, the media client could at the same time generate queryfingerprints of the channel feed that the media client receives throughthe broadcast infrastructure. And the media client could compare thequery fingerprints with the reference fingerprints in order to findfingerprint matches with sufficient certainty. As discussed above, oneor more such match points could be used as a basis to establishsynchronous lock, mapping client time with reference time for instance.

Further, as the metadata service receives the timestamped metadata, themetadata service provides the timestamped metadata to the media client.And in line with the discussion above, the media client could then usethe timestamps of the reference fingerprints and the timestamps of themetadata as a basis to reunite the metadata with the frames of thechannel feed with which the metadata was correlated when it was in-band.Namely, for each received instance of metadata, the media client couldidentify a received reference fingerprint that has the same timestamp asthe received instance of metadata, and the media client could identifythe query fingerprint that the media client deemed to match thatidentified reference fingerprint. Based on this analysis, the mediaclient could therefore conclude that the received instance of metadatacorresponds with the channel-feed frame represented by the identifiedquery fingerprint. And the media client could accordingly use themetadata based on its correlation with that particular frame, in any ofthe ways discussed above, among other possibilities.

FIG. 3 depicts an implementation where the capture server correlateseach instance of metadata with the associated reference fingerprint byattaching the instance of metadata with the reference fingerprint, suchas by including them in a common data record for instance. Thisimplementation is shown including an ACR service, which as noted abovecould be a server application executed on a server, possibly integratedwith the capture server. With this example arrangement, as the captureserver generates the reference fingerprints with attached metadata, thecapture server provides the reference fingerprints with attachedmetadata to the ACR service.

In this example arrangement, as the ACR service receives the referencefingerprints with attached metadata, the ACR service provides thetimestamped reference fingerprints with attached metadata to the mediaclient. And here too, in line with the discussion above, the mediaclient could at the same time generate query fingerprints of the channelfeed that the media client receives through the broadcast infrastructureand could compare the query fingerprints with the reference fingerprintsin order to find fingerprint matches with sufficient certainty, possiblyfor purposes of establishing synchronous lock or the like.

Here, as the media client finds a fingerprint match between a queryfingerprint and a reference fingerprint with attached metadata, themedia client can reunite the metadata with the channel feed framerepresented by the query fingerprint as discussed above. And the mediaclient could accordingly use the metadata based on its correlation withthat particular frame, in any of the ways discussed above, among otherpossibilities.

In these implementations, the capture server, ACR service, and possiblymetadata service could all form part of a computing system for carryingout the operations described in cooperation with the media client.Further, as discussed above, the computing system might first identifythe channel at issue, such as by searching reference data for referencefingerprints that match client-provided query fingerprints. And as alsodiscussed above, the computing system might require, as a conditionprecedent for carrying out the above operations, that the media clientis not receiving in-band metadata, which the computing system mightdetermine in the manner discussed above for instance.

FIG. 4 is next a simplified block diagram of a computing system thatcould operate in accordance with the present disclosure. As shown inFIG. 4 , the example computing system includes a network communicationinterface 400, a processing unit 402, and non-transitory data storage404, any or all of which could be integrated together or, as shown,communicatively linked together by a system bus, nets pork, or otherconnection mechanism 406.

The network communication interface 400 could comprise one or morenetwork connection mechanisms to facilitate communication with one ormore other entities. Each such network communication interface couldcomprise a wireless or wired Ethernet interface or other type of networkinterface, for engaging in IP communication and/or other type of networkcommunication, among other possibilities.

The processing unit 402 could comprise one or more general purposeprocessors (e.g., microprocessors) and/or one or more specializedprocessors (e.g., application specific integrated circuits). And thenon-transitory data storage 404 could comprise one or more volatileand/or non-volatile storage components, such as optical, magnetic, orflash storage for instance. Further, as shown, the data storage 404 ofthe example computing system stores or otherwise embodies programinstructions 408. These program instructions could be executable by theprocessing unit 402 to carry out (e.g., to cause the computing system tocarry out) various operations described herein.

In line with the discussion above, a representative example of such acomputing system could include various modules, such as variousservices, servers, program modules, or the like, configured to carry out(e.g., to cause the computing system to carry out) variouscomputing-system operations.

For instance, the computing system could include a capture module thatis configured to generate reference fingerprint data representing theframe of the media stream and to extract the in-band metadata from thechannel feed. Further, the computing system could include one or moremodules configured to transmit out-of-band to the media client thegenerated reference fingerprint data in correlation with the extractedmetadata, where the correlation of the extracted metadata with thereference fingerprint data representing the frame of the media stream isuseable by the media client as a basis to re-correlate the metadata withthe frame of the media stream, so as to enable the media client to carryout an action based on the correlation of the metadata with the frame ofthe media stream.

Various features discussed herein can be implemented in this context aswell, and vice versa.

FIG. 5 is a simplified block diagram of an example media client,operable in accordance with the present disclosure. This media devicecould take various forms as discussed above, such as a contentpresentation device 106 and/or a receiver 104, and it could also be acombination of multiple such devices, perhaps a content presentationdevice having an integrated receiver, among other possibilities.

As shown in FIG. 5 , the example media client could include a contentcommunication interface 500, a content presentation interface 502, anetwork communication interface 504, a processing unit 506, andnon-transitory data storage 508, any or all of which could be integratedtogether or, as shown, communicatively linked together by a system bus,network, or other connection mechanism 510.

Content communication interface 500 could comprise a physicalcommunication interface for receiving and/or outputting media content,such as a media stream. As such, the content communication interfacecould include one or more wired and/or wireless interfaces forestablishing communication with and receiving media content in analog ordigital through a media-distribution path and, in some implementations,for outputting received media content in analog or digital form forreceipt by an associated entity. Thus, the content communicationinterface might include interfaces such as coaxial cable connections,antennas, receivers, tuners, or the like, for receiving media contentfrom a content distributor, and local communication interfaces compliantwith protocols such as DVI, HDMI, VGA, USB, BLUETOOTH, WIFI, amongnumerous others for outputting media content to associated devices.Other examples could also be possible.

The content presentation interface 502, which might be included in anend-user content presentation device, could comprise one or morecomponents to facilitate presentation of the received media content, Byway of example, the content presentation interface could comprise a userinterface such as a display screen and/or a loudspeaker, as well as oneor more drivers or other components for processing the received mediacontent to facilitate presentation of the content on the user interface.

The network communication interface 504 could comprise a networkconnection mechanism to facilitate out-of-band communication on anetwork, and/or for otherwise engaging in direct or networkedcommunication with one or more other local or remote entities. As such,the network communication interface could comprise a wireless or wiredEthernet interface or other type of network interface, for engaging inIP communication and/or other type of network communication.

The processing unit 506 could then comprise one or more general purposeprocessors (e.g., microprocessors) and/or one or more specializedprocessors e.g., application specific integrated circuits). And thenon-transitory data storage 508 could comprise one or more volatileand/or non-volatile storage components, such as optical, magnetic, orflash storage. Further, the data storage 508 stores program instructions512, which could be executable by processing unit 506 to carry out(e.g., to cause the media client to carry out) various operationsdescribed herein.

FIG. 6 is next a flow chart depicting an example method that could becarried out in accordance with the present disclosure. This method couldbe carried out in a communication system in which a channel feedcarrying a media stream is broadcast for delivery of the media stream toa media client, and where the channel feed further includes metadata inband and correlated with a content frame of the media stream, but wherethe metadata gets removed from the channel feed before the media streamreaches the media client.

As shown in FIG. 6 , at block 600, the method includes a computingsystem extracting the metadata from the channel feed that carries themedia stream and that includes the metadata in-band and correlated witha content frame of the media stream. And at block 602, the methodincludes the computing system transmitting, in an out-of-band message tothe media client, the extracted metadata in correlation with referencefingerprint data that represents the frame of the media stream, thecorrelation being useable by the media client as a basis to re-correlatethe metadata with the frame of the media stream, so as to enable themedia client to carry out an action based on the correlation of themetadata with the frame of the media stream.

In line with the discussion above, the method could additionally involvethe computing system determining that the media client does not receiverepresentative metadata with the broadcast channel feed. And the methodcould involve the computing system carrying out the extracting andtransmitting responsive to the determining that the media client doesnot receive the representative metadata with the broadcast channel feed.

As further discussed above, the act of extracting the metadata from thechannel feed could involve extracting the metadata from a copy of thechannel feed, the copy being provided before the metadata gets removedfrom the channel feed. For instance, the channel feed could be providedby a broadcaster, and the computing system could receive the copy of thechannel feed from the broadcaster.

As further discussed above, the media stream could take various forms.For instance, the media stream could comprise a television channel.

Further, as discussed above, the channel feed could include the metadatain-band in various ways. For instance, the channel feed could includethe metadata carried in a transport stream that also carries the mediastream, and/or the channel feed could include the metadata in the mediastream itself, such as watermarked in the media stream and/or, if themedia stream includes video frames, carried in a vertical ancillary dataspace of a video frame of the media stream, among other possibilities.

Still further, as discussed above, the act of transmitting to the mediaclient the extracted metadata in correlation with reference fingerprintdata representing the frame of the media stream could involvetransmitting to the media client the extracted metadata timestamped witha reference timestamp and transmitting to the media client the referencefingerprint data timestamped with the same reference timestamp.Alternatively or additionally, the act of transmitting to the mediaclient the extracted metadata in correlation with reference fingerprintdata representing the frame of the media stream could involvetransmitting to the media client the extracted metadata attached withthe reference fingerprint data.

Yet further, as discussed above, the media client could use thecorrelation of the extracted metadata with the reference fingerprintdata representing the frame of the media stream as a basis tore-correlate the metadata with the frame of the media stream, so as toenable the media client to carry out a content-related action based onthe correlation of the metadata with the frame of the media stream. Andthe content-related action could take various forms, such as but notlimited to dynamic ad insertion.

Where the present disclosure refers to individual devices, servers,modules, or other entities, the disclosure also contemplates thepossibility of each such entity being a single entity or being acombination of multiple entities. For instance, a given entity couldcomprise a platform or system including multiple devices configured tointerwork together or to operate separately or in another manner.

The present disclosure also contemplates one or more computer readablemedia being encoded with, storing, or otherwise embodying programinstructions executable by a processing unit (e.g., one or moreprocessors) to carry out various described operations.

Exemplary embodiments have been described above. Those skilled in theart will understand, however, that changes and modifications may be madeto these embodiments without departing from the true scope and spirit ofthe invention.

What is claimed is:
 1. In a communication system in which a channel teedcarrying a media stream is broadcast for delivery of the media stream toa media client, wherein the channel feed further includes metadatain-band and correlated with a content frame of the media stream, butwherein the metadata gets removed from the channel feed before the mediastream reaches the media client, a method comprising: extracting, by acomputing system, the metadata from the channel feed; and transmitting,by the computing system, in an out-of-band message to the media client,the extracted metadata in correlation with reference fingerprint datarepresenting the frame of the media stream therein the correlation ofthe extracted metadata with the reference fingerprint data representingthe frame of the media stream is useable by the media client as a basisto re-correlate the metadata with the frame of the media stream, so asto enable the media client to carry out an action based on thecorrelation of the metadata with the frame of the media stream.
 2. Themethod of claim 1, further comprising the computing system determiningthat the media client does not receive representative metadata with thebroadcast channel feed.
 3. The method of claim 2, further comprising thecomputing system carrying out the extracting and transmitting responsiveto the determining that the media client does not receive therepresentative metadata with the broadcast channel feed.
 4. The methodof claim 1, wherein extracting the metadata from the channel feedcomprises extracting the metadata from a copy of the channel feed, thecopy being provided before the metadata gets removed from the channelfeed.
 5. The method of claim 4, wherein the channel feed is provided bya broadcaster and wherein the computing system receives the copy of thechannel feed from the broadcaster.
 6. The method of claim 1, wherein themedia stream comprises a television channel.
 7. The method of claim 1,wherein the channel feed including the metadata in-band comprises thechannel feed including the metadata carried in a transport stream thatalso carries the media stream, and wherein extracting the metadata fromthe channel feed comprises reading the metadata from the transportstream.
 8. The method of claim 1, wherein the channel feed including themetadata in-band comprises the channel feed including the metadata inwatermarking in the media stream, and wherein extracting the metadatafrom the channel feed comprises reading the metadata from thewatermarking.
 9. The method of claim 1, wherein the channel feedincluding the metadata in-band comprises the channel feed including themetadata carried in a vertical ancillary data space of a video frame ofthe media stream, and wherein extracting the metadata from the channelfeed comprises reading the metadata from the vertical ancillary dataspace of the video frame.
 10. The method of claim 1, whereintransmitting to the media client the extracted metadata in correlationwith reference fingerprint data representing the frame of the mediastream comprises transmitting to the media client the extracted metadatatimestamped with a reference timestamp and transmitting to the mediaclient the reference fingerprint data timestamped with the samereference timestamp.
 11. The method of claim 1, wherein transmitting tothe media client the extracted metadata in correlation with referencefingerprint data representing the frame of the media stream comprisestransmitting to the media client the extracted metadata attached withthe reference fingerprint data.
 12. The method of claim 1, wherein theaction carried out by the media client based on the correlation of themetadata with the frame of the media stream comprises a content-relatedaction, and wherein the media client uses the correlation of theextracted metadata with the reference fingerprint data representing theframe of the media stream as a basis to re-correlate the metadata withthe frame of the media stream, so as to enable the media client to carryout the content-related action based on the correlation of the metadatawith the frame of the media stream.
 13. The method of claim 1, whereinthe action carried out by the media client based on the correlation ofthe metadata with the frame of the media stream comprises dynamic adinsertion.
 14. In a system in which a channel feed carrying a mediastream is broadcast for delivery of the media stream to a media client,wherein the channel feed further includes metadata in-band andcorrelated with a content frame of the media stream, but wherein themetadata gets removed from the channel feed before the media streamreaches the media client, a computing system comprising: a capturemodule configured to generate reference fingerprint data representingthe frame of the media stream and to extract the metadata from thechannel feed; and one or more modules configured to transmit out-of-bandto the media client the generated reference fingerprint data incorrelation with the extracted metadata, wherein the correlation of theextracted metadata with the reference fingerprint data representing theframe of the media stream is useable by the media client as a basis tore-correlate the metadata with the frame of the media stream, so as toenable the media client to carry out an action based on the correlationof the metadata with the frame of the media stream.
 15. The computingsystem of claim 14, wherein the computing system is configured todetermine that the media client does not receive representative metadatawith the broadcast channel feed, and wherein the computing system isconfigured to carry out the extracting and transmitting responsive tothe determining that the media client does not receive therepresentative metadata with the broadcast channel feed.
 16. Thecomputing system of claim 14, wherein the channel feed including themetadata in-band comprises the channel feed including the metadatacarried in a transport stream that also carries the media stream, andwherein extracting the metadata from the channel feed comprises readingthe metadata from the transport stream.
 17. The computing system ofclaim 14, wherein the channel feed including the metadata in-bandcomprises the channel feed including the metadata in watermarking in themedia stream, and wherein extracting the metadata from the channel feedcomprises reading the metadata from the watermarking.
 18. The computingsystem of claim 14, wherein transmitting to the media client theextracted metadata in correlation with reference fingerprint datarepresenting the frame of the media stream comprises transmitting to themedia client the extracted metadata timestamped with a referencetimestamp and transmitting to the media client the reference fingerprintdata timestamped with the same reference timestamp.
 19. The computingsystem of claim 14, wherein transmitting to the media client theextracted metadata in correlation with reference fingerprint datarepresenting the frame of the media stream comprises transmitting to themedia client the extracted metadata attached with the referencefingerprint data.
 20. The computing system of claim 14, wherein theaction carried out by the media client based on the correlation of themetadata with the frame of the media stream comprises dynamic adinsertion.